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Preface 


In  August,  2002,  a  group  of  atmospheric  science  researchers  participated  in  an  Army-Scale 
Meteorology  Workshop  at  White  Sands  Missile  Range,  NM.  The  U.S.  Army  Research 
Laboratory  Battlefield  Environment  Division  sponsored  the  workshop  to  obtain  input  on  where 
the  Army  should  invest  its  atmospheric  research  resources.  One  of  the  participants  emphasized 
that  the  Army  should  have  information  relating  to  forecast  uncertainty. 

Some  Army  decision  makers  receive  weather  intelligence  from  an  Air  Force  Staff  Weather 
Officer,  who  may  use  meteorological  expertise  to  provide  a  range  of  values  or  probability 
information  relating  to  the  weather  forecast  parameters  of  interest.  Often,  however,  users  must 
rely  on  the  single  deterministic  value  provided  by  a  model  such  as  the  Pennsylvania  State 
University/National  Center  for  Atmospheric  Research  Mesoscale  Model  version  5  (MM5).  The 
Integrated  Weather  Effects  Decision  Aid  (IWEDA)  provides  automated  red/amber/green 
guidance  on  weather  impacts  on  specific  military  systems  based  solely  on  the  deterministic 
model  values.  Commanders  may  be  able  to  make  better  tactical  decisions  if  they  are  provided 
information  on  the  amount  of  certainty  associated  with  a  particular  forecast. 

Several  organizations  are  making  forecast  uncertainty  calculations  based  on  running  ensembles 
of  forecast  models.  However,  ensembles  are  not  being  run  in  support  of  battlefield  operations. 

In  response  to  the  workshop  recommendation,  a  limited  study  was  performed  to  associate 
forecast  model  uncertainty  with  forecast  accuracy. 

The  author  acknowledges  the  contributions  of  Dr.  Teizi  Henmi,  who  set  up  the  MM5  model  and 
observation  domain  over  Utah  for  a  previous  study.  The  existing  results  from  that  study  included 
many  forecasts  of  basic  surface  weather  parameters.  These  forecasts  were  validated  against 
observations  to  quantify  the  occurrences  of  different  error  amounts.  This  report  documents  the 
forecast  errors  seen,  with  the  partially  addressed  goal  of  relating  the  forecast  errors  to  model 
uncertainty  in  a  way  useful  to  Army  decision  makers. 
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Summary 


The  meteorological  community  is  recognizing  the  need  for  incorporating  weather  forecast 
uncertainty  along  with  the  forecasts  provided  to  users.  A  consensus  has  formed  that  a  forecast 
range  or  a  confidence  level  associated  with  a  single  forecast  value  can  lead  users  to  make  better 
decisions.  On  the  battlefield,  weather  intelligence  and  decision  aids  are  frequently  automated, 
based  on  a  single  deterministic  value  produced  by  a  numerical  weather  forecast  model. 
Researchers  are  investigating  various  approaches  to  determine  and  portray  weather  forecast 
uncertainty.  However,  some  of  these  approaches  are  not  currently  viable  for  use  on  the 
battlefield  because  of  the  amount  of  computing  power  or  data  communications  required.  This 
study  investigated  weather  forecast  uncertainty  in  a  compilation  of  forecast  errors  in  the  Penn 
State/National  Center  for  Atmospheric  Research  fifth-generation  Mesoscale  Model  (MM5). 

MM5  forecast  outputs  were  compared  to  observations  of  basic  surface  weather  parameters  at  50- 
70  Utah  MesoWest  sites.  The  forecasts  covered  24  hourly  forecasts  for  each  of  50  days  during 
the  winter  of  2002.  The  frequency  of  occurrence  of  the  resulting  error  amounts  was  tabulated 
and  is  presented  in  this  report.  Some  effort  was  included  to  relate  changes  in  error  frequencies  to 
readily  available  information  on  the  battlefield,  such  as  the  month,  the  forecast  time,  the 
observed  value,  the  forecast  value,  and  the  station  elevation. 

These  relationships  either  did  not  appear  to  be  significant,  or  the  limited  time  and  space  of  the 
data  included  do  not  allow  for  useful  generalizations  to  be  made.  The  error  data  presented  may 
be  useful  in  relaying  information  to  MM5  users  on  the  general  model  accuracy  seen  for  one 
specific  and  difficult  forecast  location.  Emphasis  is  given  to  the  percentage  of  forecasts  meeting 
the  U.S.  Army’s  stated  accuracy  requirements  for  temperature,  dew-point  temperature,  wind 
speed,  and  wind  direction.  The  results  for  the  percentage  of  forecasts  that  were  correct  for  all  the 
times  and  stations  combined  were:  25  percent  for  temperature  within  1°  C;  24  percent  for  dew¬ 
point  temperature  within  1°  C;  73  percent  for  wind  speed  within  2.5  m/s;  and  7  percent  for  wind 
direction  within  5°.  The  percentages  given  for  correct  wind  speed  and  wind  direction  forecasts 
may  be  misleading.  Users  will  often  want  wind  speed  forecasts  to  be  more  accurate  than  plus  or 
minus  2.5  m/s.  On  the  other  hand,  wind  direction  forecasts  will  often  be  acceptable  at  less 
stringent  limits  than  plus  or  minus  5°. 

This  report  presents  a  simple  approach  for  using  historical  error  amounts  to  modify  the  single 
forecast  value  generated  by  the  model  to  a  range  of  values.  This  can  be  accomplished  by  a  user 
specifying  the  level  of  confidence  desired  in  a  particular  instance  or  by  a  human  forecaster 
tagging  a  specific  day’s  model  output  with  an  estimate  of  high,  medium,  or  low  confidence. 
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1.  Introduction 


Weather  forecast  users  understand  that  the  forecast  may  be  wrong.  When  decisions  must  be 
made  using  imperfect  weather  information,  it  would  be  helpful  to  know  the  amount  of 
uncertainty  associated  with  the  predicted  weather.  The  meteorological  community  is  attempting 
to  address  this  goal  through  a  variety  of  approaches,  including: 

•  Categorizing  forecast  confidence  levels  based  on  specific  situations  and  experience. 

•  Providing  a  probability  or  range  forecast  rather  than  a  single  deterministic  value. 

•  Running  an  ensemble  of  separate  model  outputs  using  different  models  or  different  initial 
conditions  for  a  single  model. 

An  expert  weather  forecaster  may  be  familiar  with  a  particular  model’s  bias  and  error  tendencies, 
but  these  trends  often  pertain  only  to  a  particular  place  and  time.  Many  users  do  not  have  any 
quantitative  knowledge  about  probable  forecast  model  errors. 

On  the  battlefield,  the  primary  weather  forecast  model  used  in  support  of  Army  operations  is  the 
Penn  State/National  Center  for  Atmospheric  Research  fifth-generation  Mesoscale  Model  (MM5) 
(1).  This  study  quantifies  the  MM5  forecast  errors  seen  in  an  existing  data  set  generated  in  a 
previous  nowcasting  research  effort  (2).  The  results  cannot  be  generalized  to  other  times  or 
locations  based  on  this  limited  analysis,  but  they  do  provide  a  single  set  of  quantified  forecast 
error  occurrences. 
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2.  Methodology 


2,1  MM5  Model  Runs 

Each  MM5  model  run  was  initialized  with  30  hours  of  Global  Foreeast  System  (GFS)  model 
output  data  (5).  The  initial  6  hours  served  as  spin-up  time  prior  to  the  24  hourly  foreeast 
validation  times.  The  MM5  was  then  run  in  3  nests  on  the  Army  High  Performanee  Computing 
Resouree  Center’s  Cray  eomputer; 

•  The  outer  nest  eontained  55  x55  grid  points  at  45  km  grid  point  spaeing  for  a  2430  x  2430 
km  domain. 

•  The  middle  nest  eonsisted  of  55  x  55  grid  points  at  15  km  grid  point  spaeing,  eovering  a 
810x810  km  domain. 

•  The  inner  nest  ineluded  85  x  85  grid  points  at  5  km  grid  point  spaeing,  resulting  in  a  420  x 
420  km  domain. 

Hourly  output  from  the  inner  nest  was  saved  eaeh  day  for  OIZ  through  OOZ,  equating  to  6  p.m. 
through  5  p.m.  the  following  day,  loeal  Utah  time. 

The  area  of  the  inner  nest  is  shown  by  the  green  square  in  figure  1 .  The  red  dots  indieate 
loeations  registered  by  the  Utah  MesoWest  eooperative  as  having  surfaee  observing  stations,  but 
many  of  the  loeations  do  not  report  the  required  data  on  the  hour  to  be  used  for  these  evaluations 
(4).  The  smaller  blue  reetangles  indieate  areas  with  appropriate  stations  reporting  at  the 
beginning  of  the  study. 
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MM5  inner  nest 


Figure  1.  Map  of  MM5  Inner  Nest  Model  Domain  Over  Northern  Utah. 

MM5  forecasts  of  temperature,  dew-point  temperature,  and  u-  and  v-wind  components  at  the  grid 
points  were  bilinearly  interpolated  to  the  surface  station  locations.  Well  over  100  different 
stations  within  the  MM5  domain  reported  observations  on  some  of  the  various  days  covered. 
Typically,  however,  approximately  50-70  stations  would  be  available  at  a  forecast  time. 

The  model  forecast  runs  were  performed  beginning  mid  December  2002  through  February  2003 
for  days  when  the  GFS  model  and  the  surface  observations  were  available.  During  this  period,  a 
total  of  50  days  was  used,  including  8  in  December,  18  in  January,  and  24  in  February. 

2,2  F orecast  Accuracy  Analysis 

The  forecast  value  of  the  basic  weather  parameter  provided  by  the  MM5  model  was  compared  to 
the  observed  value.  Since  the  ultimate  goal  is  to  be  able  to  provide  information  beneficial  to 
Army  decision  makers,  the  error  ranges  were  generally  grouped  relative  to  the  accuracy 
requirement  stated  by  Army  intelligence  experts  (5).  The  desired  forecast  accuracy  for  both 
temperature  and  dew-point  temperature  is  within  1°C.  An  analysis  of  relative  humidity  might  be 
more  meaningful  to  many  users  than  dew-point  temperature,  but  the  surface  observation  stations 
used  in  the  study  reported  dew-point  temperature  and  not  relative  humidity.  The  Army  has 
specified  that  wind  speed  forecasts  are  needed  within  an  accuracy  of  5  kts.  An  accuracy 
requirement  stated  as  a  percentage  of  the  actual  wind  speed  value  would  often  reflect  a  more 
realistic  specification  than  the  static  value  given,  but  for  this  analysis  the  value  was  simply 
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rounded  to  2.5  m/s.  Wind  direction  data  were  omitted  for  any  observation  with  wind  speed  less 
than  1  m/s  to  reduce  the  misleading  errors  associated  with  light  and  variable  winds.  The  Army’s 
wind  direction  accuracy  requirement  of  5°  is  highlighted  in  this  analysis,  but  in  general  a  more 
reasonable  range  of  45°  is  included. 

One  of  the  aims  in  investigating  weather  forecast  uncertainty  is  to  be  able  to  provide  information 
to  decision  makers  on  when  the  forecast  is  expected  to  be  more  or  less  accurate.  The  data  set 
available  for  this  study  did  not  contain  information  on  the  actual  weather  synoptic  situations,  and 
no  attempt  has  been  made  here  to  associate  forecast  errors  with  specific  weather  scenarios.  A  list 
of  readily  available  factors  that  might  be  associated  with  forecast  error  amounts  was  devised,  to 
include: 

•  month 

•  forecast  time 

•  observed  value  of  parameter 

•  forecast  value  of  parameter 

•  station  elevation 

The  following  section  will  provide  the  forecast  errors  seen  as  a  function  of  these  variables, 
although  some  are  covered  only  briefly. 


3.  Results 


3,1  Forecast  Errors  Associated  with  the  Month 

All  of  the  model  runs  were  performed  during  a  single  winter,  with  8  days  in  December,  1 8  in 
January,  and  24  in  February.  Basic  error  statistics  are  provided  separately  for  each  of  these  three 
months  for  temperature  and  dew-point  temperature  in  table  1  and  for  wind  speed  and  wind 
direction  in  table  2. 
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Table  1.  Forecast  Error  Statistics  by  Month  for  Temperature  and  Dew-Point  Temperature. 


TEMPERATURl 

3(°C) 

DEW-POINT  ( 

:°c) 

OBS 

BIAS 

ABS 

ERR 

OBS 

BIAS 

ABS 

ERR 

DEC 

AVG 

-3.1 

I.O 

2.3 

-7.5 

1.8 

2.9 

STDEV 

5.4 

2.7 

1.8 

5.0 

3.7 

3.0 

MAX 

II.7 

12.6 

12.6 

3.9 

25.4 

25.4 

MIN 

-23.3 

-8.0 

0 

-35.2 

-lO.I 

0 

COUNT 

12000 

12000 

JAN 

AVG 

0.5 

2.1 

3.0 

-5.3 

2.4 

3.4 

STDEV 

4.9 

3.3 

2.6 

5.9 

4.4 

3.7 

MAX 

15.0 

29.5 

29.5 

15.0 

50.1 

50.1 

MIN 

-23.0 

-17.0 

0 

-51. 1 

-13.2 

0 

COUNT 

28000 

28000 

FEB 

AVG 

-2.1 

I.O 

2.5 

-7.1 

1. 1 

2.9 

STDEV 

6.0 

3.2 

2.2 

6.1 

4.1 

3.1 

MAX 

14.8 

42.5 

42.5 

II.8 

65.3 

65.3 

MIN 

-35.6 

-13.6 

0 

-59.1 

-25.4 

0 

COUNT 

37000 

37000 

The  left  eolumn  labeled  OBS  under  eaeh  weather  parameter  eontains  statisties  of  the  observed 
data  highlighting  the  large  range  of  values  seen,  ineluding  some  erroneous  observations.  Some 
bad  observations  were  deleted  before  running  the  statisties,  but  other  bad  data  points  were  not 
found  until  after  all  the  statistieal  analyses  had  been  caleulated.  The  small  number  of  ineorrect 
reported  observations  should  not  have  a  signifieant  impaet  on  the  average  error  values  based  on 
many  thousands  of  validations.  The  temperature  and  dew-point  temperature  foreeasts  refleet  a 
warm  bias  in  eaeh  month,  but  by  differing  amounts.  The  average  absolute  error  values  are 
slightly  higher  in  January  than  in  Deeember  or  February. 


6 


Table  2.  Forecast  Error  Statistics  by  Month  for  Wind  Speed  and  Wind  Direction. 


WIN 

D  SPEED ( 

'm/s) 

WIND] 

DIRECTION  (deg) 

OBS 

BIAS 

ABS 

ERR 

OBS 

BIAS 

ABS 

ERR 

DEC 

AVG 

3.5 

I.O 

2.4 

183 

6 

47 

STDEV 

3.0 

2.9 

2.0 

91 

63 

43 

MAX 

19.8 

II.9 

12.3 

359 

180 

180 

MIN 

0 

-12.3 

0 

0 

-180 

0 

COUNT 

1 1000 

8000 

JAN 

AVG 

2.2 

0.4 

1.7 

187 

9 

61 

STDEV 

1.8 

2.2 

1.5 

102 

78 

49 

MAX 

14.5 

14.8 

14.8 

359 

180 

180 

MIN 

0 

-10.5 

0 

0 

-180 

0 

COUNT 

24000 

16000 

FEB 

AVG 

2.7 

0.3 

1.8 

190 

10 

61 

STDEV 

2.4 

2.4 

1.6 

105 

77 

49 

MAX 

23.6 

II.4 

14.6 

359 

180 

180 

MIN 

0 

-14.6 

0 

0 

-180 

0 

COUNT 

32000 

24000 

The  number  of  validations  for  wind  direction  totals  about  one-quarter  to  one-third  the  number  for 
wind  speed.  This  drop  reflects  the  large  percentage  of  wind  observations  with  speed  less  than  1 
m/s,  which  were  not  used  in  the  wind  direction  statistics.  The  predominance  of  light  wind  speeds 
contributed  to  relatively  low  wind  speed  errors.  The  average  errors  in  wind  speed  and  wind 
direction  were  virtually  identical  in  the  January  and  February  results. 

Comparing  the  forecast  accuracies  across  these  three  specific  months  shows  that  there  can  be 
significant  differences  in  the  bias  or  absolute  error  occurrences.  It  is  impossible  to  draw  any 
conclusions  about  long-term  error  trends  associated  with  a  particular  month  from  this  limited 
study.  It  seems  logical  that  any  change  in  systemic  biases  might  be  related  to  different  seasons, 
but  with  data  from  only  a  single  season  in  a  single  year,  no  further  investigations  relating  to 
monthly  differences  would  be  worthwhile. 

3.2  Forecast  Errors  Associated  with  the  Forecast  Time 

Another  factor  potentially  associated  with  varying  forecast  error  amounts  is  the  time  of  day. 

Since  each  of  the  forecast  model  runs  was  initialized  at  OIZ  (6  p.m.  local  time),  the  following 
results  cannot  treat  the  time  of  day  independently  from  the  lag  time  of  the  hourly  forecasts  from 
1-24  hours.  In  spite  of  this  limitation,  the  December  forecasts  were  analyzed  by  time  of  day  by 
grouping  the  results  in  six  4-hour  intervals.  The  average  forecast  errors  were  calculated  for  the 
following  time  groups: 
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•  evening  (forecast  hours  1-4,  equating  to  6  p.m.  -  9  p.m.  local  time) 

•  nightl  (forecast  hours  5-8,  10  p.m.  -  1  a.m.) 

•  night2  (forecast  hours  9-12,  2  a.m.  -  5  a.m.) 

•  morning  (forecast  hours  13-16,  6  a.m.  -  9  a.m.) 

•  midday  (forecast  hours  17-20,  10  a.m.  -  1  p.m.) 

•  afternoon  (forecast  hours  21-24,  2  p.m.  -  5  p.m.) 

Figure  2  shows  the  results  for  these  time  periods  for  temperature  forecasts  from  the  8  days  in 
December.  Points  on  the  graph  correspond  to  the  percentage  of  the  forecasts  falling  within  the 
error  category  listed  on  the  x-axis.  The  categories  are  bands  covering  2°  C  error  ranges, 
extending  from  an  underforecast  of  7  to  9“  at  the  left  side  of  the  chart  to  an  overforecast  of  1 1  to 
13°  at  the  right  side.  The  midday  and  afternoon  forecasts  contain  slightly  less  of  a  warm  bias 
than  the  other  times,  but  the  differences  between  the  different  groups  of  forecast  times  are  not 
very  large.  The  percentage  of  the  forecasts  meeting  the  Army’s  stated  requirement  of 
temperature  forecasts  within  plus  or  minus  1°  C  ranges  from  32  percent  of  the  evening  forecasts 
to  27  percent  of  the  afternoon  forecasts. 


Temperatures 

All  December  Forecast  Hours  1-24 


40% 


S  30% 


O  20% 

o 


10% 


0% 


Forecast  Error  amount 

M - Underforecast  - ►  M - Overforecast  - ► 


Figure  2.  Percentage  of  Temperature  Forecasts  with  Specified  Error  Amounts  by  Time  of  Day. 

A  similar  chart  for  wind  speed  forecast  errors  is  provided  in  figure  3.  The  documented  Army 
requirement  for  wind  speed  forecasts  within  plus  or  minus  2.5  m/s  is  very  broad,  particularly  in 
low  wind  cases.  Rather  than  use  consistent  5  m/s  error  bands  providing  minimal  information, 
these  results  show  2.5  m/s  error  ranges.  The  percentage  of  forecasts  meeting  the  Army  goal  for 
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accuracy  can  still  easily  be  determined  by  adding  the  -2.5  to  0  m/s  error  occurrences  with  the  0  to 
+2.5  m/s  errors.  The  morning  forecasts  were  most  likely  to  contain  wind  speeds  higher  than 
those  actually  observed.  This  bias  was  prevalent  throughout  all  the  time  periods,  based  on  the 
generally  low  wind  speeds  observed.  The  percentage  of  forecasts  within  the  required  accuracy 
varied  from  58  percent  in  the  midday  forecasts  to  68  percent  in  the  evening  forecasts. 


Wind  Speeds 

All  December  Forecast  Hours  1-24 


50% 


,  40% 


=  30% 

u 

O 

**- 

O 

■£  20% 


10% 


0% 


Forecast  Error  amount 

- Underforecast  - ^  A -  Overforecast  - ► 


Figure  3.  Percentage  of  Wind  Speed  Forecasts  with  Specified  Error  Amounts  by  Time  of  Day. 

While  the  wind  speed  accuracy  requirement  of  plus  or  minus  2.5  m/s  may  be  broader  than 
desired,  the  wind  direction  forecast  goal  of  plus  or  minus  5°  seems  exceedingly  stringent.  This 
study  incorporates  error  bands  covering  45°,  but  the  errors  falling  within  the  -5  to  +5°  range  are 
totaled  separately,  in  order  to  specify  the  percentage  of  forecasts  actually  meeting  the 
requirement  as  it  stands.  Note  that  in  figure  4,  the  x-axis  values  to  the  left  and  right  of  this  -5  to 
5°  point  incorporate  ranges  of  40“  each,  so  summing  the  errors  falling  within  these  three  ranges 
gives  the  percentage  of  forecasts  accurate  within  plus  or  minus  45°.  The  remaining  points 
include  45°  ranges,  where  of  course  an  error  of -180°  is  identical  to  an  error  of +180°.  The  wind 
direction  error  in  these  cases  exhibits  a  slightly  greater  bias  in  the  counterclockwise  direction  in 
the  evening  and  in  the  clockwise  direction  by  the  afternoon  forecast. 
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Wind  Directions 

All  December  Forecast  Hours  1-24 


Counterclockwise 


Forecast  Error  amount 
-►  ◄ -  Clockwise 


Figure  4.  Percentage  of  Wind  Direction  Forecasts  with  Specified  Error  Amounts  by  Time  of  Day. 

One  aspect  to  the  question,  “How  good  is  the  forecast  model?”  is  how  often  it  meets  the  required 
accuracy.  This  is  certainly  too  simplistic,  especially  since  the  required  accuracy  depends  on  the 
situation  rather  than  being  a  constant  fixed  value.  However,  since  this  study  easily  provides  the 
data  on  how  often  the  required  accuracy  is  satisfied,  several  charts  highlighting  this  information 
are  included. 

Figure  5  summarizes  this  information  by  forecast  time.  The  wind  speed  forecasts  show  a 
significant  decrease  in  percent  accuracy  from  the  evening  forecasts  to  the  morning  through 
afternoon  forecasts.  The  temperature  forecasts  reflect  a  small  decrease  in  accuracy  by  the 
midday  forecasts,  although  the  dew-point  temperature  forecasts  are  accurate  slightly  more  often 
at  those  times.  The  wind  direction  forecasts  that  fall  within  5°  of  the  observed  wind  direction  are 
depicted  by  the  light  green  circles,  while  those  within  45“  are  the  dark  green  circles  specified  as 
WD+  in  the  legend.  The  45°  criterion  is  met  most  frequently  during  the  morning  and  midday 
forecast  times. 
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Forecast  Accuracy  by  Time  December 


evening  nighti  night2  morning  midday  afternoon 

Forecast  Hours 


♦  Wind  Spd 
WS  Avg. 
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-  -  Temp  Avg. 
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Forecast  Hour  1 
occurs  at  6  p.m. 
local  time 


*T&DP:  +  1°C  WS:+2.5m/s  WD:  +5  deg  WD+: +45  deg 


Figure  5.  Percentage  of  Forecasts  Satisfying  Required  Accuracy  by  Forecast  Time. 

These  results  do  not  show  consistent  differences  in  the  forecast  accuracy  between  the  various 
times  of  day.  Some  of  the  apparent  differences  may  be  attributed  to  the  model  start  time  rather 
than  the  relative  capability  of  the  model  to  forecast  for  different  times  of  the  day  or  night. 
However,  a  drop-off  in  forecast  accuracy  from  the  1-hour  to  the  24-hour  forecasts  is  not  evident 
to  the  extent  expected.  Since  the  results  did  not  appear  promising  for  providing  useful 
information  to  decision  makers,  this  analysis  by  forecast  time  was  not  performed  for  the  entire 
data  set  after  acquiring  the  January  and  February  data. 

3,3  Forecast  Errors  Associated  with  the  Observed  Value  of  the  Parameter 

Another  analysis  considered  with  the  initial  December  results  involved  correlating  temperature 
forecast  errors  with  the  observed  values.  Figure  6  reveals  these  errors  as  a  well-defined  function 
of  the  actual  temperature  value  reported. 
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Figure  6.  Percentage  of  Temperature  Forecasts  with  Specified  Error  Amounts  by  Observed 
Temperature  Value. 

Although  the  number  of  oases  with  observed  temperatures  in  the  most  extreme  oategories  is 
small,  the  trend  for  the  ooldest  temperatures  to  be  overforecast  and  for  the  hottest  temperatures  to 
be  underforecast  is  very  clear.  Figure  7  shows  this  substantial  drop  in  the  percent  of  forecasts 
satisfying  the  plus  or  minus  1°  C  accuracy  requirement.  The  percent  of  forecasts  seen  as 
accurate  in  these  December  cases  was  37  percent  when  the  observed  temperature  fell  between  5 
below  and  5  above  0°  C,  but  dropped  by  10  percent  or  more  in  each  5“  observed  temperature 
range  category  moving  away  from  these  more  moderate  values. 
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Forecast  Accuracy  by  Observed  Values  December 
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Observation  Ranges  (deg  C) 

Figure  7.  Percentage  of  Forecasts  Satisfying  Required  Accuracy  by  Observed  Values. 

After  considering  these  results,  it  was  decided  that  the  approach  of  relying  on  observed  values 
would  not  be  of  much  benefit  to  users,  since  they  would  not  know  the  observed  value  ahead  of 
time.  It  is  generally  intuitive  that  extreme  cases  are  more  likely  to  be  missed,  so  no  further 
analyses  were  performed  using  this  method  for  the  other  weather  parameters  or  the  January  or 
February  data. 

3,4  Forecast  Errors  Associated  with  the  Forecast  Value  of  the  Parameter 

The  association  of  extreme  observations  with  greater  forecast  error  may  be  similar  to  the  one 
expected  for  extreme  forecast  values.  Human  weather  forecasters  generally  believe  that  “going 
out  on  a  limb”  with  a  forecast  well  outside  the  normal  values  is  usually  riskier  than  a  more 
moderate  forecast.  Numerical  weather  forecast  models  do  not  temper  calculated  values  that  fall 
far  from  climatological  or  persistence-based  expectations.  The  investigations  considering 
forecast  values  described  in  this  section  include  the  data  from  all  three  months  used  in  this  study. 

As  can  be  seen  in  figure  8,  the  bias  for  predicting  too  warm  temperatures  shows  up  even  in  the 
small  number  of  hottest  forecasts.  Incorporating  some  sort  of  bias  correction  to  these  forecasts 
was  considered,  using  the  average  bias  from  the  preceding  month.  A  quick  look  at  the  average 
bias  amounts  listed  in  table  1  indicates  that  correcting  January  temperature  and  dew-point 
temperature  forecasts  based  on  the  December  bias  amounts  would  have  decreased  the  average 
error  amount.  On  the  other  hand,  using  the  January  average  bias  to  modify  the  February 
forecasts  for  these  parameters  would  have  simply  transformed  a  1°  C  warm  bias  into  a  1°  C  cold 
bias.  Since  the  data  period  used  in  this  study  was  quite  limited,  no  bias  correction  method  was 
pursued. 
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Figure  8.  Percentage  of  Temperature  Forecasts  with  Specified  Error  Amounts  by  Forecast 
Temperature  Value. 

The  great  majority  of  wind  speed  forecasts  were  for  wind  speeds  below  2.5  m/s  or  for  wind 
speeds  between  2.5  and  5  m/s.  These  forecasts  were  predominantly  within  the  2.5  m/s  accuracy 
requirement.  Figure  9  shows  the  wind  speed  errors  associated  with  the  wind  speed  forecast 
categories.  Not  surprisingly,  the  highest  wind  speed  forecasts  were  much  more  likely  to  be  too 
high  than  too  low. 
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Figure  9.  Percentage  of  Wind  Speed  Forecasts  with  Specified  Error  Amounts  by  Forecast 
Wind  Speed  Value. 

Providing  useful  information  relating  to  general  wind  direetion  error  tendeneies  ean  be  more 
difficult  than  with  other  basic  weather  parameters.  Wind  direction  is  frequently  listed  as  one  of 
the  most  important  variables  for  Army  users.  However,  wind  direction  rules  driven  by  the 
numerical  forecast  are  not  commonly  included  in  automated  decision  aids,  since  the  rules 
typically  specify  a  cross  wind  or  some  relative  direction  rather  than  a  fixed  compass  direction. 
Figure  10  consists  of  forecast  errors  in  wind  direction  in  a  similar  fashion  as  previous  charts. 

The  eight  forecast  wind  direction  range  categories  are  labeled  by  compass  direction  name,  with 
each  category  encompassing  22.5°  on  either  side  of  the  compass  point.  The  average  bias  amount 
of  wind  direction  error  is  a  relatively  smaller  percentage  of  total  error  than  seen  in  the  other 
parameters. 

Decision  makers  may  not  be  able  to  easily  use  information  on  wind  direction  bias.  A  bar  chart  is 
provided  in  figure  1 1 ,  incorporating  only  the  December  results  as  an  example  of  an  alternate  way 
to  view  this  information.  Some  people  may  find  the  errors  stand  out  somewhat  more  clearly  than 
in  the  overlapping  line  chart,  but  the  chart  still  requires  considerable  explanation  and  study  to 
determine  the  desired  information.  In  the  bar  chart,  the  forecast  wind  direction  categories 
displayed  on  the  x-axis  are  equivalent  to  those  used  as  separate  lines  in  figure  10.  The  forecast 
error  categories  associated  with  each  individual  forecast  direction  are  portrayed  in  a  separate 
color.  However,  the  error  categories  themselves  are  not  identical,  because  in  the  bar  chart  the 
errors  are  counted  according  to  observed  compass  direction  categories  as  opposed  to  strictly  the 
number  of  degrees  difference  in  the  forecast  and  observed  wind  values.  A  small  error  could 
occasionally  result  in  a  validation  in  the  neighboring  category.  The  information  contained  in  the 
bars  in  figure  1 1  is  described  after  the  figure. 
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Figure  10.  Percentage  of  Wind  Direction  Forecasts  with  Specified  Error  Amounts  by  Forecast  Wind 
Direction  Value. 
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Figure  11.  Bar  Chart  Version  of  Percentage  of  Wind  Direction  Forecast  Error  Categories  by  Forecast  Wind 
Direction  Value  for  December. 
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•  For  each  forecast  direction,  the  percentage  of  observations  from  that  same  direction  is 
indicated  by  a  striped  bar. 

•  The  bars  to  the  immediate  left  of  the  striped  bar  for  each  specific  color  include  the  percent 
of  cases  where  the  observed  wind  was  one  45“  wind  category  counter-clockwise  to  the 
forecast  category. 

•  Likewise,  the  bars  to  the  immediate  right  of  the  striped  bars  include  the  total  cases  where 
the  observed  wind  was  one  45°  wind  category  clockwise  from  the  forecast  category. 

•  The  remaining  bars  to  the  left  and  right  within  each  group  show  the  percentage  of  time  the 
observed  wind  direction  is  from  categories  increasingly  distant  from  the  forecast  category 
in  the  counter-clockwise  and  clockwise  directions,  respectively. 

The  highest  percentages  of  accurate  forecasts  occur  when  the  forecast  is  for  wind  direction  from 
the  northwest,  south,  or  north.  When  the  forecast  wind  direction  is  north,  the  observed  wind 
cases  are  very  predominantly  from  the  north  or  northwest.  A  similar  statement  can  be  made  for 
northeast  forecast  winds,  with  the  validating  wind  most  often  from  the  northeast  or  the  adjacent 
counter-clockwise  direction  of  north.  However,  northeast  forecasts  do  not  verily  correctly  as 
often  as  north  wind  forecasts,  and  forecasts  for  east  winds  show  the  lowest  accuracy  of  any 
direction.  This  bar  chart  is  included  for  illustrative  purposes  since  it  is  only  based  on  the  eight 
forecast  days  in  December.  Another  option  for  illustrating  all  the  wind  direction  data  is  provided 
in  table  3,  which  also  includes  only  the  December  cases.  Color  coding  can  highlight  trends  for 
an  initial  inspection,  but  a  thorough  analysis  of  the  results  included  in  a  table  like  this  can  still  be 
time-consuming.  The  following  list  summarizes  a  few  conclusions  from  the  December  wind 
direction  forecasts. 

•  The  forecasts  with  an  easterly  component  were  somewhat  less  accurate. 

•  Forecasts  for  south  or  northwest  winds  verified  accurately  most  often,  but  south  winds 
were  still  forecast  more  frequently  than  they  occurred. 

•  Actual  wind  directions  show  up  opposite  from  the  direction  forecast  in  about  5  percent  of 
the  cases,  ranging  from  3  percent  when  the  forecast  direction  is  north  to  8  percent  when  the 
forecast  direction  is  southeast.  For  this  study,  no  information  is  provided  on  whether  or  not 
frontal  passages  occurred  on  the  included  dates,  potentially  leading  to  timing  errors  on 
wind  direction  shifts. 

•  The  observed  wind  direction  was  within  the  correct  or  adjacent  compass  wind  direction  the 
majority  of  the  time,  ranging  from  49  percent  of  the  forecasts  for  east  winds  to  at  least  70 
percent  of  the  cases  when  the  model  forecast  winds  were  from  the  northwest,  south,  or 
north. 


17 


Table  3.  Percent  of  the  Time  the  Wind  was  Actually  Observed  From  Each  Compass  Direction  for  Each 
Forecast  Direction. 
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0.06 

0.08 

0.07 

0.07 

0.13 

0.16 

0.28 

0.15 

1046 

10% 

NW 

0.18 

0.08 

0.07 

0.06 

0.04 

0.06 

0.15 

0.37 

617 

6% 

Number  of  Ohs 
from  the  Given 
Wind  Direction 

973 

9% 

777 

7% 

921 

9% 

1703 

16% 

2458 

23% 

1504 

14% 

1133 

11% 

1165 

11% 

10634 

<  10% 

10-20% 

For  the  given  forecast  wind  direction,  the  cell 
color  depicts  the  percent  of  time  the  observed 
wind  is  from  the  noted  direction 

20-30% 

30-40% 

Cells  enclosed  In  double  borders  Indicate  the  observed  wind  Is  from 
_  the  same  direction  as  the  forecast  wind 

n  nn  indicates  the  observed  wind  Is  from  the  opposite  direction 

as  the  forecast  wind 


Returning  to  the  complete  results  for  all  three  months,  figure  12  depicts  the  relationship  between 
the  percent  of  accurate  forecasts  and  the  forecast  value  for  temperature,  dew-point  temperature, 
wind  speed,  and  wind  direction. 
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Forecast  Accuracy  by  Forecast  Values  winter 


Wind  Spd 


Figure  12.  Percentage  of  Forecasts  Satisfying  Required  Accuracy  by  Forecast  Value. 


As  with  the  observed  values,  when  the  foreeast  value  is  for  an  infrequent  occurrence,  it  tends  to 
be  less  accurate. 

•  Obviously,  the  most  notable  trend  is  the  steep  drop  in  accuracy  as  wind  speeds  become 
higher. 

•  A  lesser  but  still  significant  difference  exists  between  wind  forecasts  for  north  to  east  winds 
being  accurate  about  10  percent  less  frequently  than  from  other  directions.  These  numbers 
are  not  equivalent  to  the  ones  presented  in  figure  1 1  or  table  3  because  those  included  only 
the  December  model  runs.  Also,  the  previous  discussions  were  based  on  discrete  compass 
direction  categories,  while  figure  12  wind  direction  accuracy  percentages  rely  on  a  straight 
calculation  of  the  difference  between  observed  wind  direction  and  forecast  wind  direction 
being  less  than  45“. 

•  Only  a  slight  impact  can  be  seen  on  the  hottest  or  coldest  forecast  temperatures  and  dew¬ 
point  temperatures. 

3,5  Forecast  Errors  Associated  with  the  Station  Elevation 

Forecast  models  are  frequently  presumed  to  have  difficulty  handling  complex  terrain.  The 
elevation  may  not  be  resolved  well  at  specific  station  locations,  and  the  scale  of  local  orographic 
effects  may  not  be  represented  by  the  model  physics.  The  northern  Utah  area  used  in  this  study 
consists  of  widely  varying  terrain,  so  it  seems  logical  to  look  for  a  correlation  between  station 
elevation  and  forecast  accuracy.  The  surface  stations  were  arbitrarily  divided  into  categories  of 
low  (below  1500  m),  middle  (1500-2200  m),  and  high  (above  2200  m). 
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The  temperature  errors  for  each  of  the  three  elevation  categories  are  shown  in  figure  13.  The 
forecast  bias  toward  too  warm  temperatures  is  most  prevalent  at  the  higher  stations. 
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Figure  13.  Percentage  of  Temperature  Forecasts  with  Specified  Error  Amounts  by  Station  Elevation. 

The  dew-point  temperature  accuracies  are  similar  to  those  for  the  temperature  forecasts,  but  the 
low  stations  show  a  greater  improvement  over  the  middle  stations,  as  seen  in  figure  14. 
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Figure  14.  Percentage  of  Dew-Point  Temperature  Forecasts  with  Specified  Error  Amounts  by  Station 
Elevation. 
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Figure  15  portrays  the  wind  speed  forecast  accuracy  by  station  elevation.  These  appear  only  a 
little  less  accurate  at  the  high  stations. 
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Figure  15.  Percentage  of  Wind  Speed  Forecasts  with  Specified  Error  Amounts  by  Station  Elevation. 


On  the  other  hand,  figure  16  reveals  wind  direction  forecasts  significantly  more  accurate  at  the 
high  stations.  Cases  with  wind  speeds  below  1  m/s  are  omitted  in  these  results,  but  it  is  still 
likely  that  greater  wind  direction  variability  at  the  lower  stations  causes  more  difficulty  in 
forecasting  this  parameter. 
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Figure  16.  Percentage  of  Wind  Direction  Forecasts  with  Specified  Error  Amounts  by  Station  Elevation. 

This  study  indicated  that  varying  forecast  error  amounts  may  be  associated  with  station 
elevation,  based  on  data  from  many  surface  stations  in  each  of  the  selected  elevation  categories 
separated  by  1500  and  2200  m  (4900  and  7200  ft).  A  check  of  the  lowest  and  highest  subset  of 
the  stations  below  1300  m  and  above  2500  m  did  not  highlight  any  great  differences  from  the 
less  stringent  definition  used  for  low  and  high  stations.  The  forecast  error  trends  as  a  function  of 
station  height  are  summarized  below: 

•  Temperatures  and  dew-point  temperatures  were  more  likely  to  be  over-forecast  at  the  high 
stations. 

•  Since  the  dew-point  error  tends  to  be  in  the  same  direction  as  the  temperature  error,  relative 
humidity  forecasts  might  not  reflect  as  much  difference  between  the  higher  and  lower 
stations. 

•  Wind  speeds  were  somewhat  less  accurate  at  the  high  stations,  but  not  by  a  lot  considering 
they  can  be  expected  to  be  stronger  with  more  room  for  error. 

•  The  wind  direction  forecasts  were  accurate  more  often  at  the  high  stations. 

3,6  Forecast  Errors  for  All  Cases 

In  this  study,  it  appears  that  consideration  of  the  station  elevation  may  provide  some  insight  into 
the  frequency  of  various  errors  in  the  MM5  forecasts.  These  differences  cannot  be  generalized  to 
other  locations  or  times.  Since  the  forecast  errors  don’t  appear  to  vary  significantly  with  several 
of  the  factors  considered,  it  seems  worthwhile  to  view  the  total  occurrence  of  the  error  amounts 
for  all  the  cases  together.  However,  it  is  still  true  that  these  findings  cannot  be  assumed  to 
represent  MM5  errors  for  other  times  or  places. 
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Figures  17  through  20  show  the  frequency  of  the  various  error  amounts  for  temperature,  dew¬ 
point  temperature,  wind  speed,  and  wind  direction,  respectively,  for  all  the  forecasts  and  stations 
combined. 


Temperatures  (deg  C) 

Winter 


Figure  17.  Percentage  of  Temperature  Forecasts  with  Specified  Error  Amounts  for  all  Stations 
and  Forecast  Times  Combined. 
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Figure  18.  Percentage  of  Dew-Point  Temperature  Forecasts  with  Specified  Error  Amounts 
for  all  Stations  and  Forecast  Times  Combined. 
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gure  19. 


Percentage  of  Wind  Speed  Forecasts  with  Specified  Error  Amounts  for  all  Stations  and 
Forecast  Times  Combined. 
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Figure  20.  Percentage  of  Wind  Direction  Forecasts  with  Specified  Error  Amounts  for  all  Stations  and 
Forecast  Times  Combined. 
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4.  Conclusions 


4,1  Relating  Error  Amounts  to  User  Forecast  Uncertainty  Calculations 

The  previous  eharts  all  take  a  fair  amount  of  study  to  determine  the  probability  of  various 
foreeast  errors.  Table  4  presents  a  simplified  method  for  users  to  ineorporate  previously 
observed  errors  into  foreeast  uneertainty  ealeulations.  A  user  would  ehoose  the  eonfidenee  level 
desired  for  the  weather  foreeast,  based  on  its  eritieality.  Then  the  single  foreeast  value  output  by 
the  MM5  model  would  be  turned  into  a  range  of  values  by  adding  and  subtraeting  the  amount 
listed  for  the  eonfidenee  level  ehosen  and  the  parameter  of  interest. 

Table  4.  User-Selected  Confidence  Desired  to  Generate  a  Forecast  Range  Based  on  the  Frequency 
of  Forecast  Errors. 


CONFIDENCE  LEVEL  TEMP  DP  TEMP  WND  SPD  WND  DIR 
DESIRED _ (°C) _ (m/s) _ (degrees) 


5% 

10% 

^ - k 

0.2 

0.4 

0.2 

0.4 

0.1 

0.2 

3 

7 

15% 

Determine 

0.6 

0.6 

0.4 

11 

20% 

a  Forecast 

0.7 

0.8 

0.5 

15 

25% 

Range  by 

0.9 

1.0 

0.6 

19 

30% 

1.1 

1.2 

0.8 

23 

35% 

Using  the 

1.4 

1.5 

0.9 

28 

40% 

MM5 

1.6 

1.7 

1.1 

33 

45% 

Forecast 

1.8 

1.9 

1.2 

38 

50% 

Value 

2.1 

2.2 

1.4 

44 

55% 

2.3 

2.4 

1.6 

51 

60% 

plus 

2.6 

2.7 

1.8 

59 

65% 

and 

2.9 

3.0 

2.1 

68 

70% 

minus 

3.3 

3.4 

2.3 

78 

75% 

3.7 

3.9 

2.6 

91 

80% 

the  amount 

4.2 

4.4 

3.0 

105 

85% 

indicated 

4.9 

5.1 

3.4 

121 

90% 

\  r 

5.7 

6.0 

4.1 

137 

95% 

7.0 

8.2 

5.0 

154 

Alternatively,  an  experieneed  staff  weather  offieer  (SWO)  eould  determine  a  high,  medium,  or 
low  eonfidenee  in  the  model  run  based  on  knowledge  of  the  weather  situation  and  the  partieular 
foreeast  involved.  Table  5  shows  values  that  eould  be  used  to  modify  the  single  foreeast  value  in 
these  eases.  The  ranges  assoeiated  with  these  high,  medium,  and  low  eonfidenee  levels  are 
similar  to  the  user’s  desired  25  pereent,  50  pereent,  and  75  pereent  eonfidenee  levels, 
respeetively. 
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Table  5.  Staff  Weather  Officer  Confidence  Assigned  to  Generate  a  Forecast  Range  Based  on  the 
Frequency  of  Forecast  Errors. 


CONFIDENCE  EEVEL 
ASSIGNED  BY  SWO 

TEMP 

rc) 

DP 

WND  SPD 
(m/s) 

WND  DIR 
(degrees) 

HIGH 

Forecast 

Value 

I 

I 

0.5 

20 

MEDIUM 

+ 

2 

2 

1.5 

45 

LOW 

Amount 
^Indicated  ^ 

4 

4 

2.5 

90 

NOTE:  Red  cells  include  ranges  greater  than  specified  in  Army  requirements. 

Bias  trends  have  been  ignored  in  the  foreeast  ranges  ealeulated  from  the  values  in  tables  4  and  5. 
Correeting  for  biases,  if  known,  would  result  in  smaller  foreeast  ranges  for  eaeh  eonfidenee 
level.  In  the  forecasts  used  for  this  study,  the  improvements  based  on  correcting  for  bias  errors 
would  be  most  pronounced  for  temperature  and  dew-point  temperature.  Cells  shaded  red  in  table 
5  highlight  forecast  ranges  larger  than  the  Army’s  stated  accuracy  requirement.  Note  that  a 
model  forecast  at  the  high  confidence  level  would  still  result  in  a  total  wind  direction  forecast 
range  of  40°.  If  users  truly  find  this  amount  of  variability  to  be  unacceptable,  they  certainly 
wouldn’t  be  satisfied  with  the  vast  majority  of  wind  direction  forecasts. 

4.2  Recommendations 

This  study  approached  weather  forecast  uncertainty  as  a  function  of  historical  model  errors. 

Many  thousands  of  MM5  forecast  values  were  compared  to  observations.  However,  this 
“history”  was  limited  in  time  to  three  months  in  a  single  winter  season,  and  in  space  to  a  single 
geographic  area  around  northern  Utah.  Even  within  this  limited  domain,  it  is  difficult  to  clearly 
identify  variables  that  can  be  associated  with  more  or  less  forecast  uncertainty.  The  most 
promising  variable  appeared  to  be  station  elevation.  Many  more  cases  would  need  to  be 
examined  over  different  times  and  locations  to  determine  if  the  forecast  errors  seen  in  this  study 
are  representative  of  overall  MM5  performance. 

The  statistics  generated  and  presented  in  this  and  similar  studies  could  be  used  to  answer  a 
general  question  of  the  typical  accuracy  of  a  forecast  model.  But  decision  makers  on  the 
battlefield  would  benefit  much  more  by  having  information  on  the  specific  uncertainty  associated 
with  the  individual  day’s  forecast.  Additional  efforts  to  quantify  model  forecast  error  will  be 
useful. 
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In  addition  to  expanding  the  time  and  space  coverage  of  model  accuracy  statistics,  they  may  be 
used  to  determine  and  correct  for  model  biases.  Other  efforts  hold  more  promise  for  providing 
insight  into  the  expected  error  on  a  particular  day.  A  possible  follow-on  study  to  the  one 
presented  in  this  paper  could  find  how  closely  errors  from  the  previous  day  or  two’s  model  runs 
correlate  to  an  individual  day’s  forecast  error  from  the  same  model.  It  seems  logical  to  assume 
that  when  a  forecast  turns  out  to  be  wrong  one  day,  users  will  have  a  lower  confidence  in  its 
accuracy  the  next  day. 

Statistics  on  the  length  or  repetitiveness  of  poorer-than-average  model  performance  may  be 
useful.  Although  model  ensembles  are  not  currently  available  on  the  battlefield,  information 
from  ensembles  may  someday  be  used  to  estimate  weather  forecast  uncertainty  in  tactical 
situations.  Methods  to  provide  uncertainty  information  to  decision  makers  will  need  to  be 
automated  and  easy  to  interpret.  Additional  information  only  has  value  if  it  leads  to  better 
outcomes. 

Both  civilian  and  military  agencies  are  pursuing  efforts  to  facilitate  better  decision  making  in  the 
face  of  weather  forecast  uncertainty.  The  U.S.  Army  Research  Laboratory  (ARL)  is  currently 
sponsoring  Small  Business  Innovation  Research  contracts  with  this  goal  in  mind.  In  addition,  the 
Department  of  Defense  (DoD)  Center  for  Geosciences/ Atmospheric  Research  at  Colorado  State 
University  will  continue  to  coordinate  with  ARL  to  address  Army  needs  in  atmospheric  research, 
including  the  use  of  model  ensembles. 
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HUNTSVILLE  AL  35812 

NCAR  LIBRARY  SERIALS  1 

NATL  CTR  FOR  ATMOS  RSCH 
PO  BOX  3000 
BOULDER  CO  80307-3000 

US  ARMY  RESEARCH  LAB  5 

AMSRD  ARL  Cl  EB  (B  SAUTER) 

WSMRNM  88005 

US  ARMY  RESEARCH  LABORATORY  1 

ATTN;  AMSRD  ARL  Cl  IS  R  (A  SMITH) 

MAIL  &  RECORDS  MGMT 
ADELPHI  MD  20783-1197 
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1 


ADMNSTR 

DEFNS  TECHL  INEO  CTR 
ATTN:  DTIC  OCP  (EEECT  CPY)  (W  SMITH) 

8725  JOHN  J  KINGMAN  RD  STE  0944 
ET  BEE  VOIR  VA  22060-6218 

US  ARMY  RESEARCH  EABORATORY  2 

AMSRD  ARE  Cl  OK  TE 

ATTN:  K  RAPKA 

2800  POWDER  MIEE  ROAD 

ADEEPHI  MD  20783-1197 


TOTAL  53 
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INTENTIONALLY  LELT  BLANK. 
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